Automatic tuning of haptic effects

ABSTRACT

A system that generates haptic effects receives input media that includes audio data, video data, or sensory data. The system automatically generates a haptic effect track from the input media. The system then applies at least one filter to the automatically generated haptic effect track to generate a tuned haptic effect track.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/661,396, filed on Mar. 18, 2015, which claims priority of ProvisionalPatent Application Ser. No. 61/968,419, filed on Mar. 21, 2014, theentire content of each of which are hereby incorporated by reference.

FIELD

One embodiment is directed generally to haptic effects, and moreparticularly, to a device that automatically generates and tunes hapticeffects.

BACKGROUND

Haptics is a tactile and force feedback technology that takes advantageof a user's sense of touch by applying haptic feedback effects (i.e.,“haptic effects”), such as forces, vibrations, and motions, to the user.Devices, such as mobile devices, touchscreen devices, and personalcomputers, can be configured to generate haptic effects. In general,calls to embedded hardware capable of generating haptic effects (such asactuators) can be programmed within an operating system (“OS”) of thedevice. These calls specify which haptic effect to play. For example,when a user interacts with the device using, for example, a button,touchscreen, lever, joystick, wheel, or some other control, the OS ofthe device can send a play command through control circuitry to theembedded hardware. The embedded hardware then produces the appropriatehaptic effect.

Devices can be configured to coordinate the output of haptic effectswith the output of other content, such as games or other media, so thatthe haptic effects are incorporated into the other content. For example,in a gaming context, when a game is developed, an audio effect developercan develop audio effects that are associated with the game andrepresent an action occurring within the game, such as machine gun fire,explosions, or car crashes. Typically, haptic effects are added to thegame late in the game development process, such as when the gamedeveloper is finishing development of the game application, or when thegame developer ports the finished game application to a new platform.This generally results in the phenomena where haptic effects are addedafter all the audio effects have been developed. Because haptic effectsare typically added so late in the process, it generally falls on thehaptic effect developer, or some other developer, to make a decisionregarding associating a haptic effect with an audio effect. Further, anaudio effect developer typically does not have input regarding aselection of an appropriate haptic effect for an audio effect. This cancontribute to a degradation of the quality of haptic effects that areultimately incorporated into the content. This quality degradation canbe a barrier to incorporating high-quality haptic effects into suchcontent.

SUMMARY

One embodiment is a system that generates haptic effects. The systemreceives input media that includes audio data, video data, or sensorydata. The system automatically generates a haptic effect track from theinput media. The system then applies at least one filter to theautomatically generated haptic effect track to generate a tuned hapticeffect track.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of a system in accordance with oneembodiment of the invention.

FIG. 2 is a block diagram of a system that receives multimedia scenes,including audio, video and any sensory related data (e.g., acceleration,rotation, speed, etc.), automatically converts the scenes related datainto haptic effects, and then tunes/filters the haptic effects inaccordance with embodiments of the invention.

FIG. 3 is a graphical illustration of blending two scenes (“Scene1” and“Scene2”) in accordance with one embodiment.

FIG. 4 illustrates computer vision (“CV”) tuning implemented by an“object of interest” filter in accordance with one embodiment.

FIG. 5 illustrates CV tuning implemented by a “background/foreground”filter in accordance with another embodiment of the invention.

FIG. 6 illustrates CV tuning implemented by a “proximity to action”filter in accordance with another embodiment of the invention.

FIG. 7 is a flow diagram of the functionality of the system of FIG. 1when automatically tuning/filtering a haptic effect track in accordancewith an embodiment.

DETAILED DESCRIPTION

One embodiment is a system that automatically generates andtunes/filters one or more haptic effects given source data, such asaudio or video source data or any sensory data related to the content ofthe multimedia clip (e.g., accelerations of objects, rotations, speed,etc.). Embodiments can automatically convert received source data intohaptic information, and then perform the tuning or filtering to adjustthe automatically generated output to account for user preferences,localization (i.e., based on regional preferences), output deviceparameters, etc. The filtering can also be applied to haptic effectsthat have been authored. Therefore, the quality of the haptic effects isimproved.

FIG. 1 illustrates a block diagram of a system 10 in accordance with oneembodiment of the invention. In one embodiment, system 10 is part of adevice, and system 10 provides an automatic haptic effect tuningfunctionality for the device. In another embodiment, system 10 isseparate from the device, and remotely provides the automatic hapticeffect tuning functionality for the device. Although shown as a singlesystem, the functionality of system 10 can be implemented as adistributed system. System 10 includes a bus 12 or other communicationmechanism for communicating information, and a processor 22 coupled tobus 12 for processing information. Processor 22 may be any type ofgeneral or specific purpose processor. System 10 further includes amemory 14 for storing information and instructions to be executed byprocessor 22. Memory 14 can be comprised of any combination of randomaccess memory (“RAM”), read only memory (“ROM”), static storage such asa magnetic or optical disk, or any other type of computer-readablemedium.

A computer-readable medium may be any available medium that can beaccessed by processor 22 and may include both a volatile and nonvolatilemedium, a removable and non-removable medium, a communication medium,and a storage medium. A communication medium may include computerreadable instructions, data structures, program modules or other data ina modulated data signal such as a carrier wave or other transportmechanism, and may include any other form of an information deliverymedium known in the art. A storage medium may include RAM, flash memory,ROM, erasable programmable read-only memory (“EPROM”), electricallyerasable programmable read-only memory (“EEPROM”), registers, hard disk,a removable disk, a compact disk read-only memory (“CD-ROM”), or anyother form of a storage medium known in the art.

In one embodiment, memory 14 stores software modules that providefunctionality when executed by processor 22. The modules include anoperating system 15 that provides operating system functionality forsystem 10, as well as the rest of a device in one embodiment. Themodules further include an automatic haptic effect tuning module 16 thatautomatically tunes a haptic effect, as disclosed in more detail below.In certain embodiments, automatic haptic effect tuning module 16 caninclude multiple modules, where each individual module provides specificindividual functionality for automatically tuning a haptic effect. Forexample, automatic haptic effect tuning module 16 may include aconversion module that receives media comprising at least one of audioor sensory data, and automatically generates a haptic effect track fromthe media. Automatic haptic effect tuning module 16 may also include afiltering module that applies at least one selectable filter to theautomatically generated haptic effect track to generate a tuned hapticeffect track. System 10 will typically include one or more additionalapplication modules 18 to include additional functionality, such as the“Integrator” application by Immersion Corp., which automaticallygenerates corresponding haptic effects from audio/visual input.

System 10, in embodiments that transmit and/or receive data from remotesources, further includes a communication device 20, such as a networkinterface card, to provide mobile wireless network communication, suchas infrared, radio, Wi-Fi, or cellular network communication. In otherembodiments, communication device 20 provides a wired networkconnection, such as an Ethernet connection or a modem.

Processor 22 is further coupled via bus 12 to a display 24, such as aLiquid Crystal Display (“LCD”), for displaying a graphicalrepresentation or user interface to a user. Display 24 may be atouch-sensitive input device, such as a touchscreen, configured to sendand receive signals from processor 22, and may be a multi-touchtouchscreen. Processor 22 may be further coupled to a keyboard or cursorcontrol 28 that allows a user to interact with system 10, such as amouse or a stylus.

System 10, in one embodiment, further includes an actuator 26. Processor22 may transmit a haptic signal associated with a generated hapticeffect to actuator 26, which in turn outputs haptic effects such asvibrotactile haptic effects, electrostatic friction haptic effects, ordeformation haptic effects. Actuator 26 includes an actuator drivecircuit. Actuator 26 may be, for example, an electric motor, anelectro-magnetic actuator, a voice coil, a shape memory alloy, anelectro-active polymer, a solenoid, an eccentric rotating mass motor(“ERM”), a linear resonant actuator (“LRA”), a piezoelectric actuator, ahigh bandwidth actuator, an electroactive polymer (“EAP”) actuator, anelectrostatic friction display, or an ultrasonic vibration generator. Inalternate embodiments, system 10 can include one or more additionalactuators, in addition to actuator 26 (not illustrated in FIG. 1).Actuator 26 is an example of a haptic output device, where a hapticoutput device is a device configured to output haptic effects, such asvibrotactile haptic effects, electrostatic friction haptic effects, ordeformation haptic effects, in response to a drive signal. In alternateembodiments, actuator 26 can be replaced by some other type of hapticoutput device. Further, in other alternate embodiments, system 10 maynot include actuator 26, and a separate device from system 10 includesan actuator, or other haptic output device, that generates the hapticeffects, and system 10 sends generated haptic effect signals to thatdevice through communication device 20.

System 10 can further be operatively coupled to a database 30, wheredatabase 30 can be configured to store data used by modules 16 and 18.Database 30 can be an operational database, an analytical database, adata warehouse, a distributed database, an end-user database, anexternal database, a navigational database, an in-memory database, adocument-oriented database, a real-time database, a relational database,an object-oriented database, or any other database known in the art.

In one embodiment, system 10 further includes one or more speakers 32.Processor 22 may transmit an audio signal to speaker 32, which in turnoutputs audio effects. Speaker 32 may be, for example, a dynamicloudspeaker, an electrodynamic loudspeaker, a piezoelectric loudspeaker,a magnetostrictive loudspeaker, an electrostatic loudspeaker, a ribbonand planar magnetic loudspeaker, a bending wave loudspeaker, a flatpanel loudspeaker, a heil air motion transducer, a plasma arc speaker,and a digital loudspeaker.

System 10, in one embodiment, further includes a sensor 34. Sensor 34can be configured to detect a form of energy, or other physicalproperty, such as, but not limited to, acceleration, bio signals,distance, flow, force/pressure/strain/bend, humidity, linear position,orientation/inclination, radio frequency, rotary position, rotaryvelocity, manipulation of a switch, temperature, vibration, or visiblelight intensity. Sensor 34 can further be configured to convert thedetected energy, or other physical property, into an electrical signal,or any signal that represents virtual sensor information. Sensor 34 canbe any device, such as, but not limited to, an accelerometer, anelectrocardiogram, an electroencephalogram, an electromyograph, anelectrooculogram, an electropalatograph, a galvanic skin responsesensor, a capacitive sensor, a hall effect sensor, an infrared sensor,an ultrasonic sensor, a pressure sensor, a fiber optic sensor, a flexionsensor (or bend sensor), a force-sensitive resistor, a load cell, aLuSense CPS2 155, a miniature pressure transducer, a piezo sensor, astrain gage, a hygrometer, a linear position touch sensor, a linearpotentiometer (or slider), a linear variable differential transformer, acompass, an inclinometer, a magnetic tag (or radio frequencyidentification tag), a rotary encoder, a rotary potentiometer, agyroscope, an on-off switch, a temperature sensor (such as athermometer, thermocouple, resistance temperature detector, thermistor,or temperature-transducing integrated circuit), microphone, photometer,altimeter, bio monitor, or a light-dependent resistor.

In general, automatic haptic conversion algorithms that generate hapticeffects from sensory multimedia data, including audio and/or video data,lack the human artistic touch that can improve the quality of createdhaptic effects by tuning and tweaking. Embodiments inject some of thisexpert knowledge into the automatic conversion workflow when possible,using a variety of filtering/tuning techniques.

Embodiments can perform the tuning as part of the auto-conversionprocess, or be available as selectable filters that a user can selectafter the haptic effects have been automatically generated. In oneembodiment, the filters are applied to the haptic effect track, such asa pulse-code modulation (“PCM”) signal with samples of values −1 to +1.In one embodiment, the available filters, disclosed in more detailbelow, include: “emphasize bumps”, “emphasize last”, “scenes blending”,“dynamic range compression/expansion”, “actuator capability”, “object ofinterest”, “background/foreground” and “proximity to action” filters. Inone embodiment, some or all of the available filters can be implementedconcurrently when the haptic effects are automatically generated, sothat the output of the filters are selectable and are then “instantly”available.

FIG. 2 is a block diagram of a system that receives multimedia scenes,including audio, video and any sensory related data (e.g., acceleration,rotation, speed, etc.), automatically converts the scenes related datainto haptic effects, and then tunes/filters the haptic effects inaccordance with embodiments of the invention. In FIG. 2, audio, videoand/or sensor signals from various scenes 1-n of multimedia data areautomatically converted at 201-203 to haptic effects using known systemsand methods. One known system and method for automatic conversion isdisclosed in U.S. patent application Ser. No. 13/785,166, the disclosureof which is herein incorporated by reference. In general, an automaticconversion algorithm analyzes the sensory data or audio data signals,and identifies some features such as envelope, frequency content, peaks,noise, etc., and then creates a haptic track based on these features,

At 210, one or more filters are applied to the automatically convertedhaptic effects. The filters can be preselected and automaticallyapplied, or can be applied to the generated haptic track at any timeafter the automatic conversion. For example, a list of available filterscan be presented to a user in a pulldown menu of a user interface, andthe user can select one or more filters to apply. When multiple filtersare selected, they can be pipelined and applied in any selected orpredefined order. The filter or filters, after applied, produce a finalhaptic effects track 212.

In one embodiment, with multiple filters, the filters can be applied insequence, for example, filter1, then filter2, etc., where filter1 takesas input the original haptic track and outputs a new haptic track thatis fed to filter2 as input, and so on, until the final haptic effectstrack is output at 212. The order/sequence that the filters are appliedmay change the resultant input. For example, if the order in oneembodiment is filter1, filter2 and filter3, and the order in anotherembodiment is filter2, filter1 and filter3, the final haptic effectstrack may differ even though the same three filters are applied insequence in both examples.

In one embodiment, with multiple filters, the filters can be applied inparallel, where filter1, filter2, etc., all have as input the sameoriginal haptic track, and the output of each of the filters is then fedas inputs to a summation filter that outputs the final haptic effectstrack at 212. The summation filter can include a weighted sum schemewhere the weight of each input can be assigned by the user or assignedautomatically (i.e., pre-defined values). In another embodiment, all ofthe weights are equal (i.e., assigned automatically without the user'sintervention). For example, if there are five filters applied inparallel, each output track will be multiplied by a weight of 0.2 (=⅕)in the summation filter.

Further, in one embodiment, with multiple filters, a “hybrid” mode canbe implemented, where some filters can be applied in sequence and theoutput of multiple sequence can have filters applied to it in paralle1.For example, filter1, filter3 and filter4 can be applied in sequence toform sequence1. Filter2, filter3 and filter5 can be applied in anothersequence to form sequence2. The outputs of sequence1 and sequence2 canthen be fed to a summation filter to output the final haptic effectstrack at 212.

In embodiments, the input of any of the filters, or any sequence offilters, or any parallel application of filters, can be one of theoriginally converted haptic track, the output of any filter, the outputof any sequence of filters, or the output of any parallel application offilters.

With multiple filters, the choice of filters and the application mode(i.e., in parallel or in sequence) are based on the user's designstyle/taste and can change for portions of a media clip. For example, inthe first five minutes, filters 1 to 5 can be applied in sequence, inthe next two minutes, filters 2 and 3 can be applied in parallel, etc,Each of the filters can be applied with different parameters dependingon the user's style/taste (e.g., an advanced user mode), or can beapplied with default parameters (i.e., no user intervention).

In one embodiment, one available filter is the “emphasize bumps” filterthat gives more importance to short sharp but intense effects (i.e.,“bumps”) related to an important or relevant event by reducing themagnitude of the effects preceding it, as well as immediately after,sometimes with a short fade into the successive effect. In order to givebumps more importance, there is a need for the bumps to feel differentthan the closely preceding effects. Embodiments lower theintensity/magnitude of the effects closely preceding a “bump”.Embodiments find the bumps within the haptic track by, for example,finding the samples of the haptic track having an absolute value greaterthan a certain threshold, or the “bump threshold” (e.g., 0.9). Once a“bump” is found, embodiments look at the samples of the haptic trackrepresenting a “lookup window” of time before it (e.g., 500 ms). Inthese samples, the blocks of consecutive samples are found with absolutevalues smaller than a certain threshold, or “elimination threshold”(e.g., 0.3) and covering a time greater than a certain threshold, or“elimination duration threshold” (e.g., 50 ms). These samples are thennullified (i.e., magnitude values lowered to 0).

In one embodiment, one available filter is the “emphasize last” filter.In general, the last haptic effect in a clip/track should usually beintense in order to mark the end of the clip, and therefore it should beautomatically emphasized (e.g., higher magnitude). Therefore, the“emphasize last” filter finds the last effect in a haptic track andamplifies its magnitude. This can be achieved by multiplying the lasteffect's sample values by a gain factor greater than 1 (e.g., 1.5),Embodiments first find the sample in the haptic track representing thelast effect end (i.e., the last sample of the last effect). To do so,the last sample in the track with an absolute value bigger than acertain threshold, or “silence threshold” (e.g., 0.1) is found.Embodiments then find the start of the last effect (i.e., the firstsample of the last effect) by considering all the samples preceding thelast effect's end and finding the last block of “haptic silence” inthose samples. The last effect's start is the first sample that followsthis last “haptic silence”. A “haptic silence” is a block of consecutivehaptic track samples with absolute values smaller than a certainthreshold, or “silence threshold” (e.g., 0.1) and covering a timegreater than a certain threshold, or “silence duration threshold” (e.g.,100 ms).

In one embodiment, one available filter is the “scenes blending” filter,which is adapted to let the user feel the transition between one sceneand the next scene in a media clip. Embodiments apply a “fade out”strategy to the previous scene haptic track and/or a “fade in” strategyto the current scene haptic track. FIG. 3 is a graphical illustration ofblending two scenes (“Scene1” and “Scene2”) in accordance with oneembodiment. Embodiments emphasize or highlight the transition betweenthe different scenes in a clip by making the related (to each scene)haptic effects distinguishable. Embodiments implement a “fade out” andthen “fade in” approach to the filtering.

In one embodiment, the functionality is only applied if:

-   -   Both the haptic effects in the last window of time, or the        “blending lookup window” (e.g., 500 ms) in the first        scene/track, and in the first window of time, or the “blending        lookup window” in the following scene/track, are both intense        (i.e., haptic samples with max absolute value greater than 0.5        for example in each); or    -   If both effects are light effects (i.e., the haptic samples with        a maximum absolute value lesser than 0.5 for example in each)        but have close intensity (i.e., a maximum absolute values in        each of the two lookup windows differ by only 0.1, for example).

Further, a fade-out filter is applied to the first scene's last “lookupwindow” of haptic effects (“window1”) and a fade-in filter is applied tothe following scene's first “lookup window” of haptic effects(“window2”) as follows:

-   -   Embodiments find the maximum value in the first half of window1        and constructs a signal “signal1” identical to windowl that is        then modified in the following way: the samples from the        beginning until the maximum value's position will all have a        value =1; the following samples (from this maximum sample till        the end of signal1) are calculated in a way to have these sample        values decrease linearly from the identified maximum value until        0 (at the end of signal1). Window1 samples are then multiplied        by signal1 samples (element by element) and the result is        injected in the original haptic track instead of window1.    -   Embodiments then find the maximum value in window2 and construct        a signal “signal2” identical to window2 that is then modified in        the following way: the samples from the maximum value position        until the end of signal2 will all have a value =1; the preceding        samples (from the beginning of signal2 until this maximum        sample) are calculated in a way to have these samples values        increase linearly from 0 (at the beginning of signal2) until the        identified max value window2 samples are then multiplied by        signal2 samples (element by element) and the result is injected        in the original haptic track instead of window2.

In one embodiment, one available filter is the “dynamic rangecompression/expansion” filter that is adapted to make the weak effectsweaker and/or make the strong effects stronger. Embodiments boost themagnitude of intense effects and reduce the magnitude of small amplitudeeffects. The filter first transforms haptic sample values to alogarithmic scale (in decibels dB). All the haptic samples with valuesgreater than a certain threshold “thr1”(e.g. −30 dB) will have a newvalue calculated using a gain value “gain” (e.g., 1.5) calculated asfollows:

new value=((old_value-thr1)*gain)+thr1.

All the haptic samples with values lesser than a certain threshold“thr2” (e.g., −50 dB) will have a new value calculated using anattenuation value “att”(e.g., 2) calculated as follows:

new_value=thr2-((thr2-old_value)*att).

In one embodiment, one available filter is the “actuator capability”filter that filters the automatic generated haptic track to adapt forthe capability of the actuator(s) in the intended playback device. Ingeneral, changes are made to a haptic track by extending or shorteningsome or all of the elementary haptic effects by a certain pre-definedvalue (e.g., 10 ms), depending on the device and the quality of theactuator(s) of the playback device. For example, for a texture-basedhaptic effect, the tuning may include changing the parameter thatdefines the cycle value (i.e., the length of time the effect plays andtime where there is no effect). For example, a textured haptic effectmay include 10 ms of effect and 10ms of silence for the duration of theeffect. In one example, the cycle value for a texture-based hapticeffect on a “Galaxy S4” smartphone from Samsung Corp. was 10 ms, but ona “G2” smartphone from LG Corp., the effect was better rendered with acycle value of 25 ms, despite both having an LRA as a haptic outputdevice. For this filter, one input needed is the capabilities of theactuator(s) or other haptic output devices of the playback device,including the type of haptic output devices, the location of the hapticoutput devices, etc. This can be manually provided, or automaticallyread from pre-stored values in a look-up table.

In other embodiments, some filters can be applied to tune the generatedhaptic effects in order to haptically emphasize a specific object'srelated events in the media clip (i.e., “object of interest”), orinstead reduce the importance of these events. This object can beselected by the user/designer in one embodiment, or be automaticallydetected by the system in another. Another embodiment can also tune theeffects given the proximity of the action to the camera. In general,these embodiments are directed to improving an “out of context” problemthat sometime occurs with automatically generated haptic effects usingsensory data. More specifically, this issue can be clearly observedthrough the magnitude/intensity of the effects. For example, whenderiving haptic effects using data captured from a bike movement in aBMX activity, the automatically generated haptic effects typically havethe same magnitude when the bike is close to and when it is far from thecamera. As a result, the haptic effects feel less realistic because theydo not reflect this notion of proximity.

Further, haptic effects related to an object/action can create confusionfor the viewer in the presence of another person/object moving in thevideo with a closer distance to the camera. This confusion is due to thefact that the viewer perceives the effects as related to the “action”happening in the foreground (the closest to the camera) while in realitythey may be derived from the “action” in the background (the farthestfrom the camera). The solution in this specific case would be to derivethe displayed haptic effects from the “foreground action” if possible(i.e., if sensory data is available), and if not, reduce the magnitudeof the effects derived from the “background action”.

Because the “object of interest, “background/foreground” and “proximityto action” filters are based on video input, computer vision approachesare used in embodiments.

FIG. 4 illustrates computer vision (“CV”) tuning implemented by the“object of interest” filter in accordance with one embodiment. In FIG.4, the user at 401 chooses the object/person (i.e., the object or personconsidered of interest by the user/designer) that the haptics should berelated to by clicking on its two dimensional representation at thebeginning of the clip (assuming that sensory data was acquired for thisobject at 405). A CV algorithm will track this object at 402 in thevideo and estimate its distance to the camera in each N frames at 403.The haptic effects created for this object at 406 are then modulated (inthe N corresponding frames) at 407 to this estimated distance. Forexample, the effects can be made stronger (i.e., multiply the samplevalues by a gain factor greater than 1) when the object is close to thecamera, or made weaker (i.e., multiply the sample values by a reductionfactor less than 1) when the object is far from the camera. The value ofthe gain/reduction factor can change linearly with the estimateddistance to the camera.

The object's tracking is done by tracking visual features related tothis object in each frame and estimating its movement. The visualfeatures that can be used include (but are not limited to): color,shape, Scale Invariant Feature Transform (“SIFT”), Speeded Up RobustFeatures (“SURF”), Histogram of Oriented Gradients (“HOG”) descriptor,Haar like features, Shi-Tomasi features and wavelet transform. Thetracking can be performed using Optical Flow calculation techniques,Kalman filtering or Particle Filtering approaches.

FIG. 5 illustrates CV tuning implemented by the “background/foreground”filter in accordance with another embodiment of the invention. In FIG.5, a CV algorithm at 502 autonomously segments the objects in each scene(N frames) into foreground and background action objects/event (i.e., ifthe object is in the foreground but not moving, it is not taken intoaccount). Another algorithm identifies these objects using patternmatching approaches at 503 and generates a haptic track using thesensory data 505 related to the object identified as in the foreground,if this data is available (labeled with the object type). If not, thehaptics are created using the background object's sensory data. Thebackground/foreground filtering can also be applied dynamically.Throughout a media clip, the objects/persons can change positions andtherefore the objects/persons in the foreground can be different atdifferent moments. Embodiments can adapt to this constraint and changethe haptic effects accordingly. The haptic effects created for theidentified object/person can also be tuned based on its distance to thecamera as described above. This distance can be inferred from thetracking/identification step.

The matching approach/nature identification of an object is done bydetecting the features of the object in the image and then classifyingthem against features of well-known objects. The features that can beused include (but not limited to): color, shape, Scale Invariant FeatureTransform (“SIFT”), Speeded Up Robust Features (“SURF”), Histogram ofOriented Gradients (“HOG”) descriptor, Haar like features, Shi-Tomasifeatures, wavelet transform. The classification can be done using NeuralNetworks, Support Vector Machines (“SVM”), Adaboost classifiers, Cascadeclassifiers, K Nearest Neighbors (“KNN”) and Bayesian Networks.

FIG. 6 illustrates CV tuning implemented by the “proximity to action”filter in accordance with another embodiment of the invention. In FIG.6, a CV algorithm estimates the proportion of pixels moving fast in eachscene (N frames) at 601. If this value is high, the camera is close tothe “action” (e.g., a point-of-view (“POV”) camera) in the scene, andtherefore the generated haptic effects should have a bigger magnitude.If not, the haptic effects should feel weaker. One embodiment uses anoptical flow calculation to estimate this proximity.

FIG. 7 is a flow diagram of the functionality of system 10 of FIG. 1when automatically tuning/filtering a haptic effect track in accordancewith an embodiment. In one embodiment, the functionality of the flowdiagram of FIG. 7 is implemented by software stored in memory or othercomputer readable or tangible medium, and executed by a processor. Inother embodiments, the functionality may be performed by hardware (e.g.,through the use of an application specific integrated circuit (“ASIC”),a programmable gate array (“PGA”), a field programmable gate array(“FPGA”), etc.), or any combination of hardware and software.

At 702, a haptic effect is automatically generated in response toreceived input media that can include audio data or video data, and mayinclude sensory data for one or more objects in a scene. The result isan automated haptic effect track that may include a plurality of clipsor scenes.

At 704, the selection of one or more types of tuning or filtering forthe automatically generated haptic effect track is received. In oneembodiment, the available filters include “emphasize bumps”, “emphasizelast”, “scenes blending”, “dynamic range compression/expansion”,“actuator capability”, “object of interest”, “background/foreground” and“proximity to action” filters.

At 706, the automated generated haptic effect track is filtered based onthe selection or selections at 704, and the final filtered haptic effecttrack is generated.

As disclosed, a haptic effect track is automatically generated. Thisbaseline haptic effect track is then tuned or filtered based on aselection of one or more tuning preferences. The resultant haptic effecttrack is consequently adapted to a user's preference.

Several embodiments are specifically illustrated and/or describedherein, However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention,

What is claimed is:
 1. A method of generating haptic effects comprising:receiving input media comprising at least one of audio data, video data,or sensory data; automatically generating a haptic effect track from theinput media; and applying at least one filter to the automaticallygenerated haptic effect track to generate a tuned haptic effect track.